Computer networks, such as the Internet, private intranets, extranets and virtual private networks, are increasingly being used for a variety of endeavors including the storage and retrieval of information, communication, electronic commerce, entertainment, and other applications. In these networks, certain computers known as servers are used to store and supply information. One type of server, known as a host server, provides access to information such as data or programs stored in various computer file formats but generally referred to as a "document". Each such document is actually a highly formatted computer file containing data structures that are a repository for a variety of information including text, tables, graphic images, sounds, motion pictures, animations, computer program code, and many other types of digitized content information.
Other computers in the network, known as clients, allow a user to access a document by requesting that a copy be sent by the home server over the network to the client.
Documents are typically referenced by the client specifying an address which identifies the server that stores the document. After the user specifies a document address to the client computer, the address portion is sent over the network to a naming service in order to obtain instructions for how to establish a connection with the correct home server. Once the connection is established, the server retrieves the document from its local disk or memory storage and transmits the document over the network to the client. The network connection is then terminated.
Computer and network industry analysts and experts are presently quite concerned that traffic over large networks such as the Internet is becoming so heavy that the very nature of the way in which it is possible to use them may have to change. The present difficulties are no doubt the result of exponential increases in the number of users as well in the number of large documents such as media files being sent. As a result of this unprecedented demand in need for bandwidth and access to networks, Internet Service Providers (ISPs), backbone providers, and other carriers that provide the physical connections necessary to implement the Internet face a corresponding unprecedented demand for bandwidth. This demand exists at all levels of the network hierarchy including Points Of Presence (POPs), central access nodes, network access points, and exchange points, such as metropolitan area exchanges.
As it turns out, much of the traffic on the Internet is redundant in the sense that different users request the same documents from the same servers over and over again. Therefore, it is becoming increasingly apparent that techniques such as document caching may be deployed to reduce the demand for access. A document cache provides a way to reduce the number of repeated requests originating, from say, a given enterprise or ISP for the same document from many clients. By intercepting client requests for the same document, the cache serves copies of the original document to multiple client locations.
Using a cache, the process for providing document files to the client computers changes from the normal process. In particular, when the user of a client computer, connected to say a given enterprise or ISP, requests a document, the cache server is requested to obtain the document from the Internet. While the document is being transmitted down to the client computer, a copy is stored in the cache memory such as a disk local to the cache. Therefore, when another client computer connected to the same enterprise or ISP requests the same document, rather than requesting the document from the Internet, the request is served from the local cache. Because the redundancy rate for Internet information ranges from about 40% up to about 90%, local caching provides significant advantages. Not only is the speed of downloading apparently faster to the users of the client computers, but also the demand for backbone utilization is reduced.
Cache servers can typically be implemented as a proxy server software application running on a network appliance or other computer system that is placed physically between the client application and the document servers. The proxy server acts as a gate keeper, receiving all packets destined for the Internet, and examining them to determine if it can fulfill requests locally. However, when using proxy servers, it is typically necessary to configure the client browser, proxy server, routers, or other network infracture equipment located at an enterprise or ISP in order to redirect the request messages to the proxy server. This is problematic however, since reconfiguration of browsers is typically not possible, and even the reprogramming of routers is considered to be difficult for service providers.
Other problems are created when proxy servers are placed in the path of network traffic. In particular, the message throughput must be reduced in order to allow the proxy to examine each packet. Furthermore, proxy servers create a single point of failure whereby all of the clients connected to the proxy server lose their network access if the proxy server fails.
Therefore, proxy servers are unreliable and do not scale well as the amount of traffic increases.
Similar difficulties exist with other types of network appliances, such as firewalls, security servers, and the like, which are expected to intercept client message traffic.